Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

ABSTRACT

Systems, methods, and computer-readable media for feedback on and improving the accuracy of super-resolution imaging. In some embodiments, a low resolution image of a specimen can be obtained using a low resolution objective of a microscopy inspection system. A super-resolution image of at least a portion of the specimen can be generated from the low resolution image of the specimen using a super-resolution image simulation. Subsequently, an accuracy assessment of the super-resolution image can be identified based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier. Based on the accuracy assessment of the super-resolution image, it can be determined whether to further process the super-resolution image. The super-resolution image can be further processed if it is determined to further process the super-resolution image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.16/233,258, filed Dec. 27, 2018, which is a continuation of U.S.application Ser. No. 16/027,056, filed Jul. 3, 2018, now U.S. Pat. No.10,169,852, where the entire contents of both applications areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to providing feedback on and improvingthe accuracy of super-resolution imaging.

BACKGROUND

Inspecting materials for uniformity and detection of anomalies isimportant in disciplines ranging from manufacturing to science tobiology. Inspection often employs microscopy inspection systems toexamine and measure specimens. Specimens as used herein refer to anobject of examination (e.g., wafer, substrate, etc.) and artifact refersto a specimen, portion of a specimen, features, abnormalities and/ordefects in the specimen. For example, artifacts can be electron-based orelectronic devices such as transistors, resistors, capacitors,integrated circuits, microchips, etc., biological abnormalities, such ascancer cells, or defects in a bulk material such as cracks, scratches,chips, etc.

Microscopy inspection systems can be used to enhance what a naked eyecan see. Specifically, microscopy inspection systems can magnifyobjects, e.g. features and abnormalities, by increasing the amount ofdetail that one can see (e.g., optical resolution). Optical resolution,as used herein, refers to the smallest distance between two points on aspecimen that can still be distinguished as two separate points that arestill perceivable as separate points by a human. Optical resolution canbe influenced by the numerical aperture of an objective, among otherparameters. Typically, the higher the numerical aperture of anobjective, the better the resolution of a specimen which can be obtainedwith that objective. A single microscopy inspection system can have morethan one objective, with each objective having a different resolvingpower. Higher resolution objectives typically capture more detail thanlower resolution objectives. However, higher resolution objectives, e.g.because of their smaller field of view, typically take much longer toscan a specimen than lower resolution objectives.

To obtain higher resolution images, such as those captured according toa higher resolution objective or those created using super-resolutiontechniques, without sacrificing speed, artificial intelligence modelscan be used to infer and simulate a super-resolution image from alow-resolution image. Such methods can be achieved without actuallyscanning the specimen using a higher resolution objective but instead byusing all or a portion of a low-resolution image of a specimen, e.g.detected artifacts in a low-resolution image. These methods will bereferred to herein interchangeably as super-resolution, super-resolutionsimulation, super-resolution generation, high-resolution simulation, andthe images produced by these methods will be referred to hereininterchangeably as super-resolution images and high resolution imagesthat are simulated, e.g. using a high-resolution simulation.Super-resolution images, as used herein, can include images created atresolutions greater than the resolution limits of a microscopy system.Specifically, super-resolution images can include images at resolutionsbeyond the diffraction limit of a given microscopy system or imagescreated beyond the limits of digital image sensors of a given microscopysystem. Super-resolution images, as used herein, can also include imagessimulated within resolution limits of a given microscopy system, but ata higher resolution than a low resolution image (e.g., asuper-resolution image can be an image simulated at the highestresolution at which a microscopy system is capable of imaging).

However, not all artifacts detectable at low resolution are goodcandidates for generating accurate super-resolution images. For example,an artifact detected using low resolution magnification can correspondto many artifacts detected by high resolution magnification and withoutadditional information, which can be lacking in a low-resolution imageof the artifact, it can be impossible to generate an accuratesuper-resolution image of the low resolution image, e.g. using highresolution simulation.

Accordingly, it is desirable to provide new mechanisms for providingfeedback about which artifacts found at low resolution magnification areappropriate or inappropriate for generating super-resolution images.Further, it is desirable to improve the accuracy of generatedsuper-resolution images.

SUMMARY

Various embodiments of the disclosure are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationscan be used without parting from the spirit and scope of the disclosure.Thus, the following description and drawings are illustrative and arenot to be construed as limiting. Numerous specific details are describedto provide a thorough understanding of the disclosure. However, incertain instances, well-known or conventional details are not describedin order to avoid obscuring the description. References to one or anembodiment in the present disclosure can be references to the sameembodiment or any embodiment; and, such references mean at least one ofthe embodiments.

Reference to “one embodiment” or “an embodiment” means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the disclosure. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment,nor are separate or alternative embodiments mutually exclusive of otherembodiments. Moreover, various features are described which can beexhibited by some embodiments and not by others.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Alternative language andsynonyms can be used for any one or more of the terms discussed herein,and no special significance should be placed upon whether or not a termis elaborated or discussed herein. In some cases, synonyms for certainterms are provided. A recital of one or more synonyms does not excludethe use of other synonyms. The use of examples anywhere in thisspecification including examples of any terms discussed herein isillustrative only, and is not intended to further limit the scope andmeaning of the disclosure or of any example term. Likewise, thedisclosure is not limited to various embodiments given in thisspecification.

Without intent to limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given below. Note thattitles or subtitles can be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, technical and scientific terms used herein have themeaning as commonly understood by one of ordinary skill in the art towhich this disclosure pertains. In the case of conflict, the presentdocument, including definitions will control.

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or can be learned by practice of the herein disclosedprinciples. The features and advantages of the disclosure can berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. These and otherfeatures of the disclosure will become more fully apparent from thefollowing description and appended claims, or can be learned by thepractice of the principles set forth herein.

A method can include obtaining a low resolution image of a specimenusing a low resolution objective of a microscopy inspection system. Asuper-resolution image of at least a portion of the specimen can begenerated from the low resolution image using a super-resolutionsimulation. Further, an accuracy assessment of the generatedsuper-resolution image can be identified based on one or more degrees ofequivalence between the super-resolution image and one or more actuallyscanned high resolution images of at least a portion of one or morerelated specimens identified using a simulated image classifier. Themethod can also include determining whether to further process thesuper-resolution image based on the accuracy assessment of thesuper-resolution image. Subsequently, the super-resolution image can befurther processed if it is determined to further process thesuper-resolution image.

A system can include a microscopy inspection system for inspecting aspecimen, one or more processors, and at least one computer-readablestorage medium. The microscopy inspection system can include a lowresolution objective and a high resolution objective. Thecomputer-readable storage medium can store instructions which whenexecuted by the one or more processors cause the one or more processorsto obtain a low resolution image of a specimen using the low resolutionobjective of the microscopy inspection system. The instructions canfurther cause the one or more processors to generate a super-resolutionimage of at least a portion of the specimen from the low resolutionimage using a super-resolution simulation. Further, the instructions cancause the one or more processors to generate an accuracy assessment ofthe generated super-resolution image based on one or more degrees ofequivalence between the super-resolution image and one or more actuallyscanned high resolution images of at least a portion of one or morerelated specimens identified using a simulated image classifier. The oneor more processors can also, according to execution of the instructionsstored in the computer-readable storage medium, determine whether tofurther process the super-resolution image based on the accuracyassessment of the super-resolution image. Subsequently, thesuper-resolution image can be further processed by the one or moreprocessors if it is determined to further process the super-resolutionimage.

A non-transitory computer-readable storage medium can includeinstructions which, when executed by one or more processors, cause theone or more processors to perform operations for generating asuper-resolution image for a specimen based on a low resolution image ofthe specimen. Specifically, the instructions can cause the one or moreprocessors to receive the low resolution image of the specimen capturedby a low resolution objective of a microscopy inspection system. Theinstruction can also cause the one or more processors to generate thesuper-resolution image of at least a portion of the specimen from thelow resolution image of the specimen using a super-resolutionsimulation. Further, the instructions can cause the one or moreprocessors to identify an accuracy assessment of the super-resolutionimage based on one or more degrees of equivalence between thesuper-resolution image and one or more actually scanned high resolutionimages of at least a portion of one or more related specimens identifiedusing a simulated image classifier. The instructions can also cause theone or more processors to determine whether to further process thesuper-resolution image based on the accuracy of the super-resolutionimage. Accordingly, the instructions can also cause the one or moreprocessors to further process the super-resolution image if it isdetermined to further process the super-resolution image.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the principles briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the disclosure and are nottherefore to be considered to be limiting of its scope, the principlesherein are described and explained with additional specificity anddetail through the use of the accompanying drawings in which:

FIG. 1 illustrates an example super-resolution system for generatingsuper-resolution images.

FIG. 2A is a side view of a general configuration of a microscopyinspection system, in accordance with some embodiments of the disclosedsubject matter.

FIG. 2B is a front view of a general configuration of a microscopyinspection system, in accordance with some embodiments of the disclosedsubject matter.

FIG. 3 is a flow of an example operation for using super-resolutionimage feedback control.

FIG. 4 illustrates an example computer system for controllingsuper-resolution image generation using super-resolution image feedbackcontrol.

FIG. 5 depicts a scheme of training a suitability classifier for use inproviding super-resolution image feedback control.

FIG. 6 depicts a scheme of training a simulated image classifier for usein providing super-resolution image feedback control.

DETAILED DESCRIPTION

In accordance with some embodiments of the disclosed subject matter,mechanisms (which can include systems, methods, devices, apparatuses,etc.) for providing feedback on which artifacts found at low resolutionmagnification are suitable or unsuitable for generating super-resolutionimages, which artifacts found in super-resolution images should berescanned using a higher resolution objective, and improving theaccuracy of generated super-resolution images are provided. This type offeedback is useful, for example, to selectively employ super-resolutionfor suitable portions of a specimen, to identify problematic portions ofa specimen, both at low resolution and at high resolution, and to trainartificial intelligence models for those problematic areas to generatemore accurate super-resolution images.

As disclosed herein, in some embodiments, artificial intelligence can beused to generate super-resolution images from low resolution images,determine artifacts in a low resolution scan of a specimen that areunlikely to generate accurate super-resolution images, determine animage grade for super-resolution images and based on the image gradedetermine which artifacts need to be scanned using high resolutionmagnification. The artificial intelligence algorithms can include one ormore of the following, alone or in combination: machine learning, hiddenMarkov models; recurrent neural networks; convolutional neural networks;Bayesian symbolic methods; general adversarial networks; support vectormachines; and/or any other suitable artificial intelligence algorithm.

FIG. 1 illustrates an example super-resolution system 100 that canimplement super-resolution feedback control to microscopy inspectionsystem 110 and/or computer system 150, according to some embodiments ofthe disclosed subject matter. Super-resolution feedback control caninclude: determining after a low resolution scan of a specimen,artifacts that are unlikely to produce accurate super-resolution imagesand should be scanned at higher resolution; determining an image gradefor the super-resolution images and based on the image grade determiningwhich artifacts should be scanned at higher resolution; comparing thetotal number of artifacts to a tolerance for a similar specimen or to atolerance defined for super-resolution system 100; and/or using thehigher resolution images captured for problematic areas of a specimen totrain artificial intelligence models to generate more accuratesuper-resolution images for those problematic areas.

At a high level, the basic components of super-resolution system 100,according to some embodiments, include microscopy inspection system 110and a computer system 150. Microscopy inspection system 110 can includean illumination source 115 to provide light to a specimen, an imagingdevice 120, a stage 125, a low-resolution objective 130, a highresolution objective 132, 135, control module 140 comprising hardware,software and/or firmware.

Microscopy inspection system 110 can be implemented as part of anysuitable type of microscope. For example, in some embodiments, system110 can be implemented as part of an optical microscope that usestransmitted light or reflected light. More particularly, system 100 canbe implemented as part of the nSpec® optical microscope available fromNanotronics Imaging, Inc. of Cuyahoga Falls, OH. Microscopy inspectionsystem can also be implemented as part of confocal or two-photonexcitation microscopy.

FIGS. 2A (side view) and 2B (front view), show the general configurationof an embodiment of microscopy inspection system 110, in accordance withsome embodiments of the disclosed subject matter. According to someembodiments, microscopy inspection system 110 can include two or moreobjectives 130, 132 and 135. Objectives 130, 132 and 135 can havedifferent resolving powers. Objectives 130, 132 and 135 can also havedifferent magnification powers, and/or be configured to operate withbrightfield/darkfield microscopy, differential interference contrast(DIC) microscopy and/or any other suitable form of microscopy includingfluorescents. In some embodiments, high resolution scanning of aspecimen can be performed by using a high resolution microscope like ascanning electron microscope (SEM), a transmission electron microscope(TEM), and/or an atomic force microscope (AFM). In some embodiments, ahigh resolution microscope can be a microscope that has a magnifyingpower (e.g., 10×) two times greater than a low resolution microscopy(e.g., 5×). The objective and/or microscopy technique used to inspect aspecimen can be controlled by software, hardware, an/or firmware in someembodiments. In some embodiments, high resolution microscopy can beperformed in a separate, stand-alone system from low resolutionmicroscopy. In other embodiments, low resolution objective 130 andhigher resolution objectives 132 and 135 can reside together in amicroscopy inspection unit and be coupled to nosepiece 119.

In some embodiments, an XY translation stage can be used for stage 125.The XY translation stage can be driven by stepper motor, server motor,linear motor, piezo motor, and/or any other suitable mechanism. The XYtranslation stage can be configured to move a specimen in the X axisand/or Y axis directions under the control of any suitable controller,in some embodiments. An actuator can be used to make coarse focusadjustments of, for example, 0 to 5 mm, 0 to 10 mm, 0 to 30 mm, and/orany other suitable range(s) of distances. An actuator can also be usedin some embodiments to provide fine focus of, for example, 0 to 50 μm, 0to 100 μm, 0 to 200 μm, and/or any other suitable range(s) of distances.In some embodiments, microscopy inspection system 110 can include afocus mechanism that adjusts stage 125 in a Z direction towards and awayfrom objectives 130, 132 and 135 and/or adjusts objectives 130, 132 and135 towards and away from stage 125.

Illumination source 115 can vary by intensity, number of light sourcesused, and/or the position and angle of illumination. Light source 117can transmit light through reflected light illuminator 118 and can beused to illuminate a portion of a specimen, so that light is reflectedup through tube lens 123 to imaging device 120 (e.g., camera 122), andimaging device 120 can capture images and/or video of the specimen. Insome embodiments, the lights source used can be a white light collimatedlight-emitting diode (LED), an ultraviolet collimated LED, lasers orfluorescent light.

In some embodiments, imaging device 120 can be a camera that includes animage sensor. The image sensor can be, for example, a CCD, a CMOS imagesensor, and/or any other suitable electronic device that converts lightinto one or more electrical signals. Such electrical signals can be usedto form images and/or video of a specimen.

Different topographical imaging techniques can be used (including butnot limited to, shape-from-focus algorithms, shape-from-shadingalgorithms, photometric stereo algorithms, and Fourier ptychographymodulation algorithms) with a predefined size, number, and position ofilluminating light to generate one or more three-dimensional topographyimages of a specimen.

In some embodiments, control module 140, comprising a controller andcontroller interface, can control any settings of super-resolutionsystem 100 (e.g., illumination source 115, objectives 130, 132 and 135,stage 125, imaging device 120), as well as communications, operations(e.g., taking images, turning on and off an illumination source, movingstage 125 and/or objectives 130, 132 and 135). Control module 140 caninclude any suitable hardware (which can execute software in someembodiments), such as, for example, computers, microprocessors,microcontrollers, application specific integrated circuits (ASICs),field-programmable gate arrays (FGPAs) and digital signal processors(DSPs) (any of which can be referred to as a hardware processor),encoders, circuitry to read encoders, memory devices (including one ormore EPROMS, one or more EEPROMs, dynamic random access memory (“DRAM”),static random access memory (“SRAM”), and/or flash memory), and/or anyother suitable hardware elements. In some embodiments, individualcomponents within super-resolution system 100 can include their ownsoftware, firmware, and/or hardware to control the individual componentsand communicate with other components in super-resolution system 100.

In some embodiments, communication between the control module (e.g., thecontroller and controller interface) and the components ofsuper-resolution system 100 can use any suitable communicationtechnologies, such as analog technologies (e.g., relay logic), digitaltechnologies (e.g., RS232, ethernet, or wireless), network technologies(e.g., local area network (LAN), a wide area network (WAN), theInternet) Bluetooth technologies, Near-field communication technologies,Secure RF technologies, and/or any other suitable communicationtechnologies.

In some embodiments, operator inputs can be communicated to controlmodule 140 using any suitable input device (e.g., a keyboard, mouse orjoystick).

Computer system 150 of super-resolution system 100 can be coupled tomicroscopy inspection system 110 in any suitable manner using anysuitable communication technology, such as analog technologies (e.g.,relay logic), digital technologies (e.g., RS232, ethernet, or wireless),network technologies (e.g., local area network (LAN), a wide areanetwork (WAN), the Internet) Bluetooth technologies, Near-fieldcommunication technologies, Secure RF technologies, and/or any othersuitable communication technologies. Computer system 150, and themodules within computer system 150, can be configured to perform anumber of functions described further herein using images output bymicroscopy inspection system 110 and/or stored by computer readablemedia.

Computer system 150 can include any suitable hardware (which can executesoftware in some embodiments), such as, for example, computers,microprocessors, microcontrollers, application specific integratedcircuits (ASICs), field-programmable gate arrays (FGPAs), and digitalsignal processors (DSPs) (any of which can be referred to as a hardwareprocessor), encoders, circuitry to read encoders, memory devices(including one or more EPROMS, one or more EEPROMs, dynamic randomaccess memory (“DRAM”), static random access memory (“SRAM”), and/orflash memory), and/or any other suitable hardware elements.

Computer-readable media can be any available media that can be accessedby the computer and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media can comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digital videodisk (DVD) or other optical disk storage, magnetic cassettes, magnetictape, magnetic disk storage or other magnetic storage devices, or anyother medium which can be used to store the desired information andwhich can be accessed by the computer.

According to some embodiments, computer system 150, can include anartifact suitability analysis module 160, a super-resolution module 170,a super-resolution analysis module 180, an image assembly module 190 andan artifact comparison module 195.

FIG. 3 , with further reference to FIGS. 1, 2A, 2B, 4 , shows at a highlevel, an example of a super-resolution operation 300 usingsuper-resolution feedback control, in accordance with some embodimentsof the disclosed subject matter. In some embodiments, super-resolutionoperation 300 can use super-resolution system 100. Further detailsexplaining how each module of computer system 150 can be configured, inaccordance with some embodiments of the disclosed subject matter, willbe described in connection with FIG. 4 .

At 310, microscopy inspection system 110 can scan a specimen using lowresolution objective 130. In some embodiments, the specimen can bescanned by moving imaging device 120 and/or stage 125 in an X/Ydirection until the entire surface or a desired area of a specimen isscanned. In some embodiments, one or more areas of a specimen can bescanned by using different focus levels and moving stage 125 and/orlow-resolution objective 130 in a Z direction. Imaging device 120 cancapture and generate low resolution images of the scanned specimen.

At 320, artifact suitability analysis module 160, can use artificialintelligence algorithms and/or other suitable computer programs (asexplained further herein) to detect artifacts in the generated lowresolution image and determine their suitability for super-resolutionimaging. In some embodiments, suitability can be based oncross-correlation of an artifact to known artifacts that have beenassessed as suitable or not suitable for super-resolution imaging.Cross-correlation, as referred to herein, can be a measure of similarityof two series (e.g., two images) as a function of the displacement ofone relative to the other. More specifically, an image of an artifactbeing examined and an image of a known artifact, each represents amatrix of intensity values per pixel (0-255), and cross-correlation canspecify the value associated with how different or similar the imagesare at each pixel.

In some embodiments, suitable known artifacts can be artifacts wheresuper-resolution images were generated and those images were determinedto be high confidence super-resolution images, e.g. having a high imagegrade. Conversely, known unsuitable artifacts can be artifacts wheresuper-resolution images were generated and those images were determinedto be low confidence super-resolution images, e.g. having a low imagegrade. High confidence and low confidence super-resolution images andcorresponding image grades are further described herein.

While the techniques described herein are made with reference toidentifying whether an artifact is suitable for super-resolutionsimulation, in various embodiments, the techniques can be performedwithout determining suitability of artifacts for super-resolutionsimulation.

At 330, a high resolution objective (e.g., high resolution objective 132or 135) can scan the artifacts determined to be unsuitable by artifactsuitability analysis module 160, and imaging device 120 can capture andgenerate high resolution images of the scanned artifacts. In someembodiments, the generated high resolution images can be provided asfeedback to: artifact suitability analysis module 160 to provideadditional context data for determining the suitability of an artifactfor super-resolution imaging; super-resolution module 170 to improve itsaccuracy; and/or super-resolution analysis module 180 to provideadditional context data for determining the image grade of asuper-resolution image. The high resolution images can also be providedto image assembly module 190 for incorporation into a single coherentimage, e.g. combining one or more super-resolution images and one ormore high resolution images, of a scanned specimen.

At 340, super-resolution module 170, using one or more super-resolutionalgorithms, can generate super-resolution images for the entire specimenor just the artifacts determined to be suitable for super-resolution byartifact suitability analysis module 160.

Super-resolution analysis module 180, at 350, can receivesuper-resolution images from super-resolution module 170 and usingartificial intelligence algorithm and/or other suitable computerprograms (as explained further herein) determine an image confidence ofthe super-resolution images. As will be discussed in greater detaillater an image confidence determination of a super-resolution image caninclude a specific image confidence determination of thesuper-resolution image, whether the super-resolution image is a highconfidence super-resolution image, whether the super-resolution is a lowconfidence super-resolution image, and/or an image grade of thesuper-resolution image. An image confidence determination of asuper-resolution image, as determined by the super-resolution analysismodule 180, can correspond to a predicted accuracy, e.g. as part of anaccuracy assessment, of a super-resolution image created through asuper-resolution simulation. A predicted accuracy of a super-resolutionimage can be an estimate of how accurately a super-resolution imagecreated from a low resolution image actually represents a specimen andartifacts in the specimen. Specifically, a predicted accuracy of asuper-resolution image can be an estimate of how accurately asuper-resolution image created from a low resolution image actuallyrepresents a specimen and artifacts in the specimen as if thesuper-resolution image was created by actually scanning theartifacts/specimen using a high resolution objective or an applicablemechanism for scanning the specimen at super-resolution. For example, ifsuper-resolution analysis model 180 identifies that a simulatedsuper-resolution image accurately represents 95% of an imaged specimen,then super-resolution analysis module 180 can identify that thesuper-resolution image is a high confidence super-resolution image.

An image confidence determination of a super-resolution image, asdetermined by the super-resolution analysis module can correspond todegrees of equivalence between a super-resolution image and one or moreactually scanned high resolution images of a specimen. Specifically,super-resolution analysis module 180 can determine how closely asuper-resolution image corresponds to an actual high resolution image ofthe same or similar type of specimen/artifact to determine a confidencein the super-resolution image and a degree of equivalence between thesuper-resolution image and the high resolution image. This can be basedon cross-correlation methods. As used herein, a same or similar type ofspecimen/artifact is referred to as a related specimen/artifact. Forexample, a related specimen can include an imaged material that is thesame or similar type of material as a currently analyzed specimen. Inanother example, a related specimen to a current specimen can includethe current specimen itself. If a super-resolution image closelycorrelates to an actual high resolution image of the same or similartype of specimen/artifact, then super-resolution analysis module 180 canindicate that the super-resolution image is a high confidencesuper-resolution image. Conversely, if a super-resolution image poorlycorresponds to an actual high resolution image of the same or similartype of specimen/artifact then super-resolution analysis module 180 canindicate that the super-resolution image is a low confidencesuper-resolution image and indicate for the underlying artifact to bescanned using a high resolution objective (e.g., 132 and 135) and togenerate high resolution images (as in step 330).

At 360, image assembly module 190 can assemble and stitch together (asdescribed further herein), the received super-resolution images and theimages scanned using a high resolution objective, into a single coherentimage of a scanned specimen.

At 370, artifact comparison module 195 can receive a single coherentimage of a specimen and determine a total number of artifacts for thespecimen. The artifact comparison module 195 can compare the totalnumber with a tolerance that is typical for the type of specimen thatwas scanned, or based on a tolerance defined for super-resolution system100 (e.g., by an operator, hardware/firmware/software constraints,industry guidelines, and/or any other suitable standard).

The division of when the particular portions of operation 300 areperformed can vary, and no division or a different division is withinthe scope of the subject matter disclosed herein. Note that, in someembodiments, blocks of operation 300 can be performed at any suitabletimes. It should be understood that at least some of the portions ofoperation 300 described herein can be performed in any order or sequencenot limited to the order and sequence shown in and described inconnection with FIG. 3 , in some embodiments. Also, some portions ofprocess 200 described herein can be performed substantiallysimultaneously where appropriate or in parallel in some embodiments.Additionally, or alternatively, some portions of process 200 can beomitted in some embodiments. Operation 300 can be implemented in anysuitable hardware and/or software. For example, in some embodiments,operation 300 can be implemented in super-resolution system 100.

FIG. 4 shows the general configuration of an embodiment of computersystem 150, in accordance with some embodiments of the disclosed subjectmatter

In some embodiments, artifact suitability analysis module 160 can beconfigured to receive one or more low resolution images of a specimenfrom microscopy inspection system 110 and/or any suitable computerreadable media. In some embodiments, the low resolution images can beimages captured by imaging device 120 using low resolution objective130. In further embodiments, artifact suitability analysis module 160can be configured to detect, using computer vision, one or moreartifacts in the received image(s) and determine a suitability class foreach detected artifact. Detection of an artifact can be based on, e.g.,information from a reference design (e.g., a computer aided design (CAD)file, physical layout of a specimen, etc.), deviations from a referencedesign, and/or data about known artifacts. In some embodiments, one ormore artificial intelligence algorithm(s) can be used to determine asuitability class for each identified artifact. In some embodiments, theclass can be a binary class (e.g., “suitable” and “not suitable” forsuper-resolution imaging). In other embodiments, the class can providegreater or higher resolution distinctions of classes (e.g., a lettergrade A-F, where A denotes the best grade and where F denotes the worstgrade, or a number grade 1-100, where 1 denotes the worst grade and 100denotes the best grade).

In some embodiments, artifact suitability analyzer module 160 can applya classification algorithm to determine whether a detected artifact in alow resolution image is or is not suitable for super-resolutiongeneration. In some embodiments, the classification algorithm is firsttrained with training data to identify shared characteristics ofartifacts that are suitable for super-resolution generation and thosethat are not. In some embodiments, training data can include examples oflow resolution images of artifacts along with their assigned suitabilityclasses. In some embodiments, training data can include examples of lowresolution images of artifacts along with the image grades assigned tosuper-resolution images generated for those artifacts. In someembodiments, the classification algorithm can make inferences aboutsuitability based on an artifact's type, size, shape, composition,location on the specimen and/or any other suitable characteristic. Insome embodiments, training data can also include explicit suitabilityassignments based on a portion of a specimen that is being imaged,information from a reference design, an artifact location (i.e.,location of an artifact on a specimen), type of artifact and/or itssize, shape and/or composition.

Once the classification algorithm is trained it can be applied byartifact suitability analyzer module 160 to determine whether a detectedartifact in a low resolution image is suitable or not suitable forsuper-resolution generation.

A classifier is a function that maps an input attribute vector (e.g.,X=(X₁, X₂, X₃, X₄, X_(n))), to a confidence that the input belongs to aclass (e.g., f(x)=confidence(suitability class)). In the case ofsuitability classification, attributes can be, for example, artifact'stype, size, shape, composition, location on the specimen, referencedesign and/or any other suitable characteristic, to determine anartifact's suitability for super-resolution imaging.

A support vector machine (SVM) is an example of a classifier that can beemployed. SVM operates by finding a hypersurface in the space ofpossible inputs that attempts to split the triggering criteria from thenon-triggering events. This makes the classification correct for testingdata that is near, but not identical to training data. Directed andundirected model classification approaches can be used and include,e.g., naïve Bayes, Bayesian networks, decision trees, and probabilisticclassification models providing different patterns of independence canbe employed. Classification as used herein is also inclusive ofstatistical regression that can be utilized to develop priority models.

The disclosed subject matter can employ classifiers that are trained viageneric training data, extrinsic information (e.g., reference design,high resolution images of the same or similar type specimen (referred toherein as a ground truth high resolution image)), and/or feedback fromsuper-resolution system 100, as super-resolution operation 300progresses. For example, SVM's can be configured via a learning ortraining phase within a classifier constructor and feature selectionmodule. Thus, the classifier(s) can be used to automatically perform anumber of functions, including but not limited to the following:determining the context of an artifact (e.g., location of the artifacton a specimen, the type of specimen being inspected, similar artifactson the same or similar type specimens, a reference design, a groundtruth high resolution image), and analyzing the size, shape, compositionof the artifact to better classify the artifact in order to correctlydetermine the suitability of the artifact for super-resolution imaging.

The SVM is a parameterized function whose functional form is definedbefore training. Specifically, a SVM is a function defined by one ormore separating hyperplanes in a dimensional space of multiple orinfinite dimensions. The SVM can be trained using an applicable methodfor training a supervised learning model. Training an SVM generallyrequires a labeled training set, since the SVM will fit the functionfrom a set of examples. The training set can consist of a set of Nexamples. Each example consists of an input vector, xi, and a categorylabel, yj, which describes whether the input vector is in a category.For each category there can be one or more parameters, e.g. N freeparameters in an SVM trained with N examples, for training the SVM toform the separating hyperplanes. To train the SVM using theseparameters, a quadratic programming (QP) problem can be solved as iswell understood. Alternatively, sub-gradient descent and coordinatedescent can be used to train the SVM using these parameters. Thesetechniques may include a Sequential Minimal Optimization technique aswell as other techniques for finding/solving or otherwise training theSVM classifier using such techniques.

Further, the disclosed subject matter can be implemented usingunsupervised machine learning techniques. Specifically, confidence imagedeterminations of super-resolution images can be identified usingunsupervised learning techniques. Further, suitability of artifacts inlow resolution images in being used to form a super-resolution image canbe identified using unsupervised learning techniques. Unsupervisedlearning techniques include applicable methods for recognizing patternsin uncategorized/unlabeled data. For example, a neural network can beused to implement the disclosed subject matter through unsupervisedlearning techniques.

Referring to FIG. 5 , the diagram illustrates a scheme, in accordancewith some embodiments of the disclosed subject matter, wherein detectedartifacts 510 are classified into two classes: suitable and not suitablefor super-resolution imaging. This is just an example and a plurality ofother training sets may be employed to provide greater or higherresolution distinctions of classes (e.g., the classes can representdifferent suitability grades A, B, C, D, E and F or suitability scores).Suitability of an artifact can be a measure of a likelihood that theartifact can be used to produce all or a portion of an accuratesuper-resolution image, at 340. Specifically, suitability of an artifactcan be a measure of likelihood that a super-resolution image generated,at least in part, from a low resolution image will pass as a highconfidence super-resolution image, e.g. at 350. More specifically,suitability of an artifact can be a prediction of how closely asuper-resolution image created from a low resolution image of theartifact will correspond to an actual high resolution image of the sameor similar type of specimen/artifact. For example, if there is a 95%chance that a super-resolution image created from a low resolution imageof an artifact will correspond greatly, e.g. 90% correlation, with anactual high resolution image of a related artifact, then the artifactcan be identified as suitable for super-resolution imaging, e.g. have ahigh suitability grade of A.

The suitability classifier 520 can be trained by a group of knownartifacts 515 that represent artifacts suitable for super-resolutionimaging and a group of known artifacts 517 that represent artifacts notsuitable for super-resolution imaging. In other embodiments, suitabilityclassifier 520 can be trained by a group of known artifacts thatrepresent different suitability grades. Artifacts 510 to be analyzed canbe input into suitability classifier 520, which can output a class 530,which indicates the class that the detected artifact most likely fallsinto. Further classes (e.g., a grade) can also be added if desired. Insome embodiments, suitability classifier 520 can also output a scalarnumber 525, a suitability score, that can measure the likelihood that anartifact being analyzed falls into the class suitable forsuper-resolution imaging, if so desired, or the class not suitable forsuper-resolution imaging, for example.

The various scoring techniques, described herein, can be implementedusing linear regression modeling. For example, either or both artifactsuitability scoring and super-resolution image scoring can beimplemented using linear regression modeling. Linear regression modelingis a machine learning technique for modeling linear relationshipsbetween a dependent variable and one or more independent variables. Asimple linear regression model utilizing a single scalar prediction canbe used to perform the scoring described herein. Alternatively, amultiple linear regression model utilizing multiple predictors can beused to perform the scoring described herein.

The likelihood that, an artifact, falls into a particular class is alsorefined to as a confidence level (or a confidence interval). Confidencelevel generally refers to the specified probability of containing theparameter of the sample data on which it is based is the onlyinformation available about the value of the parameter. For example, ifa 95% confidence level is selected then it would mean that if the samepopulation is sampled on numerous occasions and confidence intervalestimates are made on each occasion, the resulting intervals wouldbracket the true population parameter in approximately 95% of the cases.An example of confidence level estimation that can be adapted for use bysuper-resolution system 100 is described by G. Papadopoulos et al.,“Confidence Estimation Methods for Neural Networks: A PracticalComparison,” ESANN 2000 proceedings—European Symposium on ArtificialNeural Networks Bruges (Belgium), 26-8 Apr. 2000 D-Facto public, ISBN2-930307-00-5, pp. 75-80, which is hereby incorporated by referenceherein in its entirety. The disclosed method is just an example and isnot intended to be limiting.

In embodiments where artifact suitability analysis module 160 determinessuitability in a non-binary manner (e.g., scoring an artifact by gradeor by number), artifact suitability analysis module 160 can beconfigured to compare the determined suitability score with anacceptable suitability tolerance for super-resolution system 100, e.g.as defined by an operator, hardware/firmware/software constraints,industry guidelines, and/or any other suitable standard. For artifactsreceiving suitability scores falling below the acceptable suitabilitytolerance for super-resolution system 100, artifact suitability analysismodule 160 can indicate for the identified artifacts to be scanned usinga higher resolution objective. For artifacts receiving suitabilityscores at or above the acceptable suitability tolerance forsuper-resolution system 100, artifact suitability analysis module 160can indicate for super-resolution images to be generated for thedetected artifacts.

The classifier can also be used to automatically adjust the acceptablesuitability tolerance used for determining suitability of an artifactfor super-resolution imaging. A feedback mechanism can provide data tothe classifier that automatically impacts the acceptable suitabilitytolerance based on historical performance data and/or improvement of oneor more underlying artificial intelligence algorithms used bysuper-resolution system 100. For example, an acceptable suitabilitytolerance can initially be set so that all detected artifacts receivinga letter grade of C and above, or a number grade of 50 and above, aredeemed suitable for super-resolution imaging. If feedback fromsuper-resolution analysis module 180 shows that a large number ofartifacts determined to be suitable ultimately yielded low confidencesuper-resolution images, then the classifier can raise the acceptablesuitability tolerance making it more difficult for artifacts to beclassified as suitable. Conversely, if feedback from super-resolutionmodule 170 shows that its model has improved and is better able togenerate super-resolution images for defects previously classified asunsuitable, then the classifier can lower the acceptable suitabilitytolerance making it easier for artifacts to be classified as suitable.

The acceptable suitability tolerance used by artifact suitabilityanalyzer module 160 to determine suitability can also be automaticallyadjusted based on the importance of a specimen and/or an area of aspecimen being examined. For example, artifact suitability analyzermodule 160 can adjust the acceptable suitability tolerance upwards forspecimens and/or areas of a specimen considered important and/or adjustthe acceptable suitability tolerance downwards for specimens and/orareas of a specimen not considered important.

Note that suitability analyzer module 160 is not restricted to employingartificial intelligence for determining suitability of an artifact forsuper-resolution imaging. In some embodiments, artifact suitabilityanalyzer module 160 can be preprogrammed to recognize suitable andunsuitable artifacts. Based on the preprogrammed data, suitabilityanalyzer module 160 can process one or more low resolution images todetermine whether the low resolution images(s) include any artifactssimilar to the preprogrammed artifacts and determine suitability basedon the suitability of the preprogrammed artifacts.

In operation, in some embodiments, the artificial intelligencealgorithms used by artifact suitability analysis module 160, can bebased on comparing characteristics of and/or context data for thedetected artifact to characteristics of and/or context data of trainingdata to generate a suitability score. For example, if a detectedartifact closely resembles an artifact from the training data thatreceived a score of A, then artifact suitability analysis module 160 canassign a similar score to the detected artifact.

In further embodiments, the artificial intelligence algorithms used byartifact suitability analysis module 160, can be based on comparingcharacteristics of and/or context data for the detected artifact tocharacteristics of and/or context data of training data that yieldedhigh confidence super-resolution images (e.g., as determined bysuper-resolution analysis module 180) to generate a suitability score.For example, artifact suitability analysis module 160 can assign a lowerscore to detected artifacts resembling training data that yielded lowconfidence super-resolution images and a higher score to detectedartifacts resembling training data that yielded high confidencesuper-resolution images.

In another embodiment, the artificial intelligence algorithms used byartifact suitability analysis module 160, can be based on comparingdetected artifacts on a specimen to artifacts in a high resolution imageof the same or similar type specimen (also referred to as the groundtruth high resolution image). If the detected artifact corresponds totwo or more artifacts in the ground truth high resolution scan, and thecontext data for the detected artifact does not provide additionalinformation, then artifact suitability analysis module 160 can assign alow suitability score. Conversely, if the detected artifact correspondsto only one artifact in the ground truth high resolution image, thenartifact suitability analysis module 160 can assign a high suitabilityscore to the detected artifact.

Artifact suitability analysis module 160 can also be configured, in someembodiments, to record the identified artifacts, their suitabilityscores and the acceptable suitability tolerance at which the analysiswas performed.

In some embodiments, super-resolution module 170 can be configured toreceive one or more low resolution images of a specimen that aredetermined to be suitable for super-resolution generation, and togenerate one or more super-resolution image(s) from the receivedimage(s). Alternatively, super-resolution module 170 can be configuredto receive one or more low resolution images of a specimen irrespectiveof whether the low resolution images are deemed actually suitable forsuper-resolution generation, and to generate one or moresuper-resolution image(s) from the received images. In some embodiments,one or more artificial intelligence algorithm(s) can be used to generateone or more super-resolution images from one or more low resolutionimages. In some embodiments, the algorithms used by super-resolutionmodule 170, can consider context date like location of the artifact onthe specimen, the type of specimen being inspected, a comparison of theartifact to other artifacts detected on the same or similar specimens, areference design, low resolution images taken at different focus levelsand/or using different lighting techniques, high resolution images takenat different focus levels and/or using different lighting techniques,etc. In further embodiments, the algorithms used by super-resolutionmodule 170, can include classifying an artifact, as well as identifyingits size, shape, composition, location on the specimen and/or any othersuitable characteristic to infer an accurate high resolution image.

Super-resolution methods employed by super-resolution module 170 caninclude, but are not limited to: interpolation, super-resolution fromlow resolution depth image frames, super-resolution through fusing depthimage and high resolution color image, example-based super-resolution,and depth image super-resolution based on edge-guided method.

Some examples of interpolation that can be adapted for use bysuper-resolution module 170 are described by: Xie, J. et al.,“Edge-guided Single Depth Image Super-resolution,” IEEE Trans. ImageProcess. 2016, 25, 428-438; Prajapati, A. et al., “Evaluation ofDifferent Image Interpolation Algorithms,” Int. J. Comput. Appl. 2012,58, 466-476; Pang, Z. et al, “An Improved Low-cost Adaptive BilinearImage Interpolation Algorithm,” In Proceedings of the 2nd InternationalConference on Green Communications and Networks, Chongqing, China, 14-16Dec. 2012; Springer: Berlin/Heidelberg, Germany, 2013; pp. 691-699;Ning, L. et al., “An Interpolation Based on Cubic InterpolationAlgorithm,” In Proceedings of the International Conference InformationComputing and Automation,” Chengdu, China, 20-22 Dec. 2007; pp.1542-1545, which are hereby incorporated by reference herein in theirentirety. The disclosed methods are just examples and are not intendedto be limiting.

Some examples of super-resolution from low resolution depth image framesthat can be adapted for use by super-resolution module 170 are describedby: Schuon, S. et al., “LidarBoost: Depth Superresolution for ToF 3DShape Scanning,” In Proceedings of the 2009 the 22nd InternationalConference on Computer Vision and Pattern Recognition, Miami, FL, USA,20-25 Jun. 2009; pp. 343-350; Rajagopalan, A. N. et al., “ResolutionEnhancement of PMD Range Maps,” In Proceedings of the Joint PatternRecognition Symposium, Munich, Germany, 10-13 Jun. 2008; Springer:Berlin/Heidelberg, Germany, 2008; pp. 304-313; Al Ismaeil, K. et al.,“Dynamic Super-resolution of Depth Sequences with Non-rigid Motions,” InProceedings of the 2013 20th IEEE International Conference on Image,Melbourne, Australia, 15-18 Sep. 2013; pp. 660-664; Gevrekci, M. et al.,“Depth Map Super-resolution,” In Proceedings of the 2011 18th IEEEInternational Conference on Image, Brussels, Belgium, 11-14 Sep. 2011;pp. 3449-3452, which are hereby incorporated by reference herein intheir entirety. The disclosed methods are just examples and are notintended to be limiting.

Some examples of super-resolution through fusing depth image and highresolution color image, that can be adapted for use by super-resolutionmodule 170 are described by: Ferstl, D. et al., “Image Guided DepthUpsampling Using Anisotropic Total Generalized Variation,” InProceedings of the IEEE International Conference on ComputernVision,Sydney, NSW, Australia, 1-8 Dec. 2013; pp. 993-1000; Yang, Q. et al.,“Spatial-Depth Super-resolution for Range Images, In Proceedings of the2007 IEEE Computer Society Conference on Computer Vision and PatternRecognition, Minneapolis, MN, USA, 17-22 Jun. 2007; pp. 1-8; Lo, K. H.et al., “Edge-Preserving Depth Map Upsampling by Joint TrilateralFilter,” IEEE Trans. Cybern. 2017, 13, 1-14, which are herebyincorporated by reference herein in their entirety. The disclosedmethods are just examples and are not intended to be limiting.

Some examples of example-based super-resolution that can be adapted foruse by super-resolution module 170 are described by: Timofte, R. et al.,“A+: Adjusted Anchored Neighborhood Regression for FastSuper-Resolution,” In Proceedings of the Asian Conference on ComputerVision, Singapore, 1-5 Nov. 2014; Springer: Cham, Switzerland, 2014; pp.111-126; Yang, J. et al., “Image Super-resolution via SparseRepresentation,” IEEE Trans. Image Process. 2010, 19, 2861-2873; Xie, J.et al., “Single Depth Image Super-resolution and Denoising via CoupledDictionary Learning with Local Constraints and Shock Filtering,” InProceedings of the 2014 IEEE International Conference on Multimedia andExpo (ICME), Chengdu, China, 14-18 Jul. 2014; pp. 1-6; Kim, J. et al.,“Accurate Image Super-resolution Using Very Deep ConvolutionalNetworks,” In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, Las Vegas, NV, USA, 27-30 Jun. 2016; pp. 1646-1654,which are hereby incorporated by reference herein in their entirety. Thedisclosed methods are just examples and are not intended to be limiting.

An example of depth image super-resolution based on edge-guided methodthat can be adapted for use by super-resolution module 170 is describedby: Zhou, D. et al., “Depth Image Super-resolution Based on Edge-GuidedMethod,” Appl. Sci. 2018, 8, 298, which is hereby incorporated byreference herein in its entirety. The disclosed method is just anexample is not intended to be limiting.

In some embodiments, an artificial intelligence algorithm used bysuper-resolution module 170 can be trained using low resolution imagesonly.

In some embodiments, super-resolution analysis module 180 can beconfigured to receive one or more super-resolution images fromsuper-resolution module 170 and/or from any computer readable media, anddetermine an image class (or grade) for each super-resolution image ofan artifact. In some embodiments, one or more artificial intelligencealgorithm(s) can be used to determine an image class forsuper-resolution images of an artifact. In some embodiments, the imageclass can be a binary class (e.g., “high confidence super-resolutionimage” and “low confidence super-resolution image”). In otherembodiments, the class can provide greater or higher resolutiondistinctions of classes (e.g., a letter grade A-F, where A denotes thebest grade and where F denotes the worst grade, or a number grade 1-100,where 1 denotes the worst grade and 100 denotes the best grade).

In some embodiments, super-resolution analysis module 180 can apply aclassification algorithm to determine an image grade for asuper-resolution image. In some embodiments, the classificationalgorithm is first trained with training data to identify sharedcharacteristics of super-resolution images of artifacts that are highconfidence super-resolution images and those that are low confidencesuper-resolution images. In some embodiments, training data can includeexamples of super-resolution images for the types of artifacts/specimensthat are being examined by super-resolution system 100 and theircorresponding image scores/grades and/or cross correspondence to actualhigh resolution images of the same or similar type of specimen/artifact.In some embodiments, the classification algorithm can make inferencesabout an image class based on a reference design, a ground truth highresolution image of the same or similar specimen type, a ground truthhigh resolution image of the same or similar artifact type, anartifact's type, size, shape, composition, location on the specimenand/or any other suitable characteristic. In some embodiments, trainingdata can also include explicit image class assignments based on aportion of a specimen that is being imaged, an artifact location (i.e.,location of an artifact on a specimen), a reference design, a groundtruth high resolution image, type of artifact and/or its size, shapeand/or composition.

Once the classification algorithm is trained it can be applied bysuper-resolution analysis module 180 to determine an image class for animage of an artifact generated by super-resolution.

A support vector machine (SVM) is an example of a classifier that can beemployed. Directed and undirected model classification approaches canalso be used and include, e.g., naïve Bayes, Bayesian networks, decisiontrees, and probabilistic classification models providing differentpatterns of independence can be employed. Classification as used hereinis also inclusive of statistical regression that can be utilized todevelop priority models.

The disclosed subject matter can employ classifiers that are trained viageneric training data, extrinsic information (e.g., reference design,ground truth high resolution image of the same or similar typespecimen/artifact), and/or feedback from super-resolution system 100, assuper-resolution operation 300 progresses. For example, SVM's can beconfigured via a learning or training phase within a classifierconstructor and feature selection module. Thus, the classifier(s) can beused to automatically perform a number of functions, including but notlimited to the following: determining context data for asuper-resolution image (e.g., location of artifact on the specimen, thetype of specimen being inspected, similar artifacts on similarspecimens, a reference design, a ground truth high resolution image ofthe same or similar type specimen, a ground truth high resolution imageof the same or similar type artifact) and analyzing the size, shape,composition of the artifact to better classify the artifact in order tocorrectly determine the image grade of a super-resolution image for anartifact.

Referring to FIG. 6 , the diagram illustrates a scheme, in accordancewith some embodiments of the disclosed subject matter, whereinsuper-resolution images of artifacts 610 are classified into twoclasses: low confidence super-resolution images and high confidencesuper-resolution images. This is just an example and a plurality ofother training sets can be employed to provide greater or higherresolution distinctions of classes (e.g., the classes can representdifferent image grades A, B, C, D, E and F or image scores 1-100). Thesimulated image classifier 620 can be trained by a group of knownsuper-resolution images 615 that are high confidence super-resolutionimages of artifacts and a group of known super-resolution images 617that represent low confidence super-resolution images of artifacts. Inother embodiments, simulated image classifier 620 can be trained by agroup of known super-resolution images that represent different imagegrades. Super-resolution images of artifacts 610 to be analyzed can beinput into simulated image classifier 620, which can output a confidenceinterval 625 that can measure the likelihood that the super-resolutionimage, being analyzed falls into a particular class (e.g., highconfidence super-resolution image and low confidence super-resolutionimage). In some embodiments, simulated image classifier 620 can alsooutput a class 630, which indicates the class that the super-resolutionimage most likely falls into. Further classes (e.g., a lettered ornumbered grade) can also be added if desired

In embodiments where super-resolution analysis module 180 determinesimage classification in a non-binary manner (e.g., scoring asuper-resolution image by grade or by number), super-resolution analysismodule 180 can be configured to compare the determined image grade withan acceptable image tolerance for super-resolution system 100, asdefined by an operator, hardware/firmware/software constraints, industryguidelines, and/or any other suitable standard. For super-resolutionimages receiving image scores falling below the acceptable imagetolerance for super-resolution system 100, super-resolution analysismodule 180 can indicate for the artifacts in the super-resolution imagesto be scanned using a higher resolution objective. Image tolerances andcorresponding image scores assigned by the super-resolution analysismodule 180 can indicate whether a super-resolution image is a highconfidence super-resolution image or a low confidence super-resolutionimage. For example, super-resolution images having image scores at orabove an acceptable image tolerance can be identified as high confidencesuper-resolution images. Conversely, super-resolution images havingimage scores below an acceptable image tolerance can be identified aslow confidence super-resolution images. For super-resolution imagesreceiving image scores at or above the acceptable image tolerance forsuper-resolution system 100, the super-resolution images can be providedto image assembly module 190.

The classifier can also be used to automatically adjust the acceptableimage tolerance used for determining whether an artifact rendered bysuper-resolution passes or fails the tolerance. A feedback mechanism canprovide data to the classifier that automatically impacts the tolerancebased on historical performance data and/or improvement of one or moreunderlying artificial intelligence algorithms used by super-resolutionsystem 100. For example, the classifier can adjust the tolerance basedon feedback about super-resolution images correctly and/or incorrectlyclassified as high or low confidence super-resolution images. Forexample, if feedback from artifact comparison module 195 shows that alarge number of super-resolution images had to be rescanned using ahigher resolution objective, then the classifier can raise theacceptable image tolerance making it more difficult for super-resolutionimages to qualify. In some embodiments, if feedback fromsuper-resolution module 170 shows that its model has improved and isbetter able to simulate high resolution images for super-resolutionimages previously classified as low confidence super-resolution images,then the classifier can lower the acceptable image tolerance making iteasier for super-resolution images to qualify.

The image tolerance used by super-resolution analysis module 180 todetermine an acceptable image tolerance can also be automaticallyadjusted based on the importance of a specimen and/or an area of aspecimen being examined. For example, super-resolution analysis module180 can adjust the acceptable image tolerance upwards for specimensand/or areas of a specimen considered important and/or adjust theacceptable image tolerance downwards for specimens and/or areas of aspecimen not considered important.

Note that super-resolution analysis module 180 is not restricted toemploying artificial intelligence for determining an image grade forsuper-resolution images. In some embodiments, super-resolution analysismodule 180 can be preprogrammed to recognize super-resolution images ofartifacts that have acceptable and non-acceptable image grades. Based onthe preprogrammed data, super-resolution analysis module 180 can processone or more super-resolution image to determine whether thesuper-resolution images(s) include any images similar to thepreprogrammed images and determine acceptable image grades based on theimage grades of the preprogrammed super-resolution and/or highresolution images.

In operation, in some embodiments, the artificial intelligencealgorithms used by super-resolution analysis module 180, can be based oncomparing characteristics of and/or context data for thesuper-resolution image to characteristics of and/or context data oftraining data to generate an image score. For example, if asuper-resolution image of an artifact closely resembles asuper-resolution image of an artifact from the training data set thatreceived an image score of A, then super-resolution analysis module 180can assign a similar score to the super-resolution image.

In another embodiment, the artificial intelligence algorithms used bysuper-resolution analysis module 180, can be based on comparingsuper-resolution images of an artifact found on a specimen to a highresolution image of the same or similar type artifact or specimen. Ifthe super-resolution analysis module 180 finds a close correspondence,then it can assign a high image score to the super-resolution image.Conversely, if super-resolution analysis module 180 finds a poorcorrespondence, then it can assign a low image score to thesuper-resolution image.

Super-resolution analysis module 180 can also be configured, in someembodiments, to record the received super-resolution images and theirimage grades, as well as the acceptable image tolerance at which theanalysis was performed.

In some embodiments, image assembly module 190 can be configured toassemble and stitch together the super-resolution images, and the actualhigh resolution images into a single coherent image of a specimen. Insome embodiments, each image of a specimen is referred to as a tile,wherein each tile can be located by its XY coordinate position in aspecimen space. For artifacts that yielded a low confidencesuper-resolution image or determined to be unsuitable forsuper-resolution imaging, and therefore, designated for scanning by ahigh resolution objective, the high resolution objective can then scanthe area on the specimen representing the tile or tiles that contain theidentified artifacts. Similarly, super-resolution module 170 cansimulate the entire tile or tiles that contain the artifacts determinedto be suitable for super-resolution imaging. The high resolution imagesof the tiles and the super-resolution images of the tiles can bestitched together based on their XY coordinate positions and/orfeature-based registration methods. This is just one example of how asingle coherent image can be assembled, and other suitable methods foraccomplishing this can be performed. In some embodiments,super-resolution module 170 can simulate the entire specimen (evenportions of the specimen that were indicated unsuitable forsuper-resolution imaging) and image assembly module 190 can replace theunsuitable portions with high resolution images of those portions. Imageassembly module 190 can use a high resolution image tile's XY location,as well as identify similar features between the high resolution imagetile and the super-resolution image tile to determine where to place thehigh resolution image tiles. Once image assembly module 190 locates thecorrect position for the high resolution image tile, thesuper-resolution image tile can be replaced with the high resolutionimage tile. While the above method assumes no more than a singleartifact per tile, the method can be adapted to accommodate multipleartifacts per tile.

In some embodiments, artifact comparison module 195 can be configured toreceive a single coherent image of a specimen (e.g., from image assemblymodule 190 and/or any suitable computer readable media) and determine atotal number of artifacts for the specimen. The artifact comparisonmodule 195 can compare the total number with a tolerance that is typicalfor the type of specimen that was scanned, or based on a tolerancedefined for super-resolution system 100, by an operator,hardware/firmware/software constraints, industry guidelines, and/or anyother suitable standard. In some embodiments, if the total number ofartifacts exceed or fall below the tolerance for the type of specimenthat was scanned, and/or the defined tolerance for super-resolutionsystem 100, then super-resolution analysis module 180, based on feedbackfrom artifact comparison module 195, can select a second set ofsuper-resolution images falling below a higher acceptable imagetolerance to be rescanned using high resolution objective 132 or 135.Specifically, super-resolution analysis module 180 can select a set ofsuper-resolution images as part of further controlling operation ofsuper-resolution system 100 to generate one or more high resolutionimages for a specimen. For example, if the acceptable image tolerancefor super-resolution analysis module 180 was initially set at 50%, andartifact comparison module 195 determines that the total number ofartifacts detected for the specimen does not seem typical for the typeof specimen examined, as explained above, then super-resolution analysismodule 180 can raise the acceptable image tolerance to 60%, and thesuper-resolution images that were assigned an image grade between 50-59%will be rescanned using a high resolution objective. Feedback tosuper-resolution analysis module 180 and adjustment to the acceptableimage tolerance can occur as many times as necessary.

In some embodiments, if the total number of artifacts exceed or fallbelow a tolerance for the type of specimen that was scanned, and/or thedefined tolerance for super-resolution system 100, then artifactsuitability analysis module 160 can select a second set of artifactsfalling below a higher acceptable suitability tolerance to be rescannedusing high resolution objective 135. Specifically, the artifactsuitability analysis module 160 can select a set of second artifacts aspart of further controlling operation of super-resolution system 100 togenerate one or more high resolution images for a specimen. For example,if the acceptable suitability tolerance for artifact suitabilityanalysis module 160 was initially set at 50%, and artifact comparisonmodule 195 determines that the total number of artifacts detected forthe specimen does not seem typical for the type of specimen examined, asexplained above, then artifact suitability analysis module 160 can raisethe suitability threshold to 60% and the artifacts that were assigned asuitability score between 50-59% will be rescanned using a highresolution objective. Feedback to artifact suitability analysis module160 and adjustment to the acceptable suitability tolerance can occur asmany times as necessary.

In some embodiments if the total number of artifacts exceed or fallbelow a tolerance for the type of specimen that was scanned and/or atolerance defined for super-resolution system 100, then artifactcomparison module 195 can determine that super-resolution module 170 isusing an unsuitable artificial intelligence model to generatesuper-resolution images and instruct super-resolution module 170 to usea different artificial intelligence model to generate super-resolutionimages for a particular specimen. Specifically, the super-resolutionmodule 170 can use different artificial intelligence models as part offurther controlling operation of super-resolution system 100 to generateone or more high resolution images for a specimen.

Although the descriptions herein refer to analyzing artifacts, themechanisms described here can also be used to analyze areas of aspecimen. For example, instead of determining suitability based onanalyzing artifacts, artifact suitability analysis module 160 candetermine suitability based on analyzing distinct areas of a specimen.Similarly, instead of determining image grades based on analyzingartifacts generated using super-resolution, super-resolution analysismodule 180 can determine image grades based on analyzing distinct areasof a specimen rendered using super-resolution.

The functionality of the components for super-resolution system 100 canbe combined into a single component or spread across several components.In some embodiments, the functionality of some of the components (e.g.,high resolution scanning by high resolution objective 132 or 135 andcomputer processing by computer system 150) can be performed remotelyfrom microscopy inspection system 110.

Note that super-resolution system 100 can include other suitablecomponents not shown. Additionally or alternatively, some of thecomponents included in super-resolution system 100 can be omitted.

In some embodiments, any suitable computer readable media can be usedfor storing instructions for performing the functions and/or processesdescribed herein. For example, in some embodiments, computer readablemedia can be transitory or non-transitory. For example, non-transitorycomputer readable media can include media such as non-transitorymagnetic media (such as hard disks, floppy disks, etc.), non-transitoryoptical media (such as compact discs, digital video discs, Blu-raydiscs, etc.), non-transitory semiconductor media (such as flash memory,electrically programmable read only memory (EPROM), electricallyerasable programmable read only memory (EEPROM), etc.), any suitablemedia that is not fleeting or devoid of any semblance of permanenceduring transmission, and/or any suitable tangible media. As anotherexample, transitory computer readable media can include signals onnetworks, in wires, conductors, optical fibers, circuits, and anysuitable media that is fleeting and devoid of any semblance ofpermanence during transmission, and/or any suitable intangible media.

The various systems, methods, and computer readable mediums describedherein can be implemented as part of a cloud network environment. Asused in this paper, a cloud-based computing system is a system thatprovides virtualized computing resources, software and/or information toclient devices. The computing resources, software and/or information canbe virtualized by maintaining centralized services and resources thatthe edge devices can access over a communication interface, such as anetwork. The cloud can provide various cloud computing services viacloud elements, such as software as a service (SaaS) (e.g.,collaboration services, email services, enterprise resource planningservices, content services, communication services, etc.),infrastructure as a service (IaaS) (e.g., security services, networkingservices, systems management services, etc.), platform as a service(PaaS) (e.g., web services, streaming services, application developmentservices, etc.), and other types of services such as desktop as aservice (DaaS), information technology management as a service (ITaaS),managed software as a service (MSaaS), mobile backend as a service(MBaaS), etc.

The provision of the examples described herein (as well as clausesphrased as “such as,” “e.g.,” “including,” and the like) should not beinterpreted as limiting the claimed subject matter to the specificexamples; rather, the examples are intended to illustrate only some ofmany possible aspects. It should also be noted that, as used herein, theterm mechanism can encompass hardware, software, firmware, or anysuitable combination thereof.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “determining,” “providing,”“identifying,” “comparing” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system memories or registersor other such information storage, transmission or display devices.

Certain aspects of the present disclosure include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present disclosurecould be embodied in software, firmware or hardware, and when embodiedin software, could be downloaded to reside on and be operated fromdifferent platforms used by real time network operating systems.

The present disclosure also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored on acomputer readable medium that can be accessed by the computer. Such acomputer program may be stored in a computer readable storage medium,such as, but is not limited to, any type of disk including floppy disks,optical disks, CD-ROMs, magnetic-optical disks, read-only memories(ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic oroptical cards, application specific integrated circuits (ASICs), or anytype of non-transient computer-readable storage medium suitable forstoring electronic instructions. Furthermore, the computers referred toin the specification may include a single processor or may bearchitectures employing multiple processor designs for increasedcomputing capability.

The algorithms and operations presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may also be used with programs in accordancewith the teachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will be apparent to those ofskill in the art, along with equivalent variations. In addition, thepresent disclosure is not described with reference to any particularprogramming language. It is appreciated that a variety of programminglanguages may be used to implement the teachings of the presentdisclosure as described herein, and any references to specific languagesare provided for disclosure of enablement and best mode of the presentdisclosure.

The super-resolution feedback control mechanism, method and system havebeen described in detail with specific reference to these illustratedembodiments. It will be apparent, however, that various modificationsand changes can be made within the spirit and scope of the disclosure asdescribed in the foregoing specification, and such modifications andchanges are to be considered equivalents and part of this disclosure.The scope of the invention is limited only by the claims that follow.

1-20. (canceled)
 21. A method, comprising: obtaining, by a computingsystem, a low resolution image of a specimen using a low resolutionobjective of a microscopy inspection system; detecting, by the computingsystem, a plurality of artifacts in the low resolution image;determining, by the computing system, that a first artifact of theplurality of artifacts is suitable for super-resolution imaging;determining, by the computing system, that a second artifact of theplurality of artifacts is not suitable for super-resolution imaging; andbased on the determining, generating, by the computing system, asuper-resolution image of a first portion of the specimen that includesthe first artifact, generating, by the computing system, a highresolution image of a second portion of the specimen that includes thesecond artifact, and generating, by the computing system, a singlecoherent image of the specimen by combining the super-resolution imageand the high resolution image.
 22. The method of claim 21, whereindetermining, by the computing system, that the first artifact of theplurality of artifacts is suitable for super-resolution imagingcomprises: cross-correlating the first artifact to known artifacts thathave been assessed as suitable for super-resolution imaging.
 23. Themethod of claim 22, wherein cross-correlating the first artifact to theknown artifacts that have been assessed as suitable for super-resolutionimaging comprises: measuring a similarity of the first artifact to eachknown artifact as a function of a displacement of the first artifactrelative to each known artifact.
 24. The method of claim 22, wherein theknown artifacts that have been assessed as suitable for super-resolutionimaging comprises artifacts where super-resolution images were generatedand determined to be high-confidence super-resolution images.
 25. Themethod of claim 21, wherein determining, by the computing system, thatthe second artifact of the plurality of artifacts is not suitable forsuper-resolution imaging comprises: cross-correlating the secondartifact to known artifacts that have been assessed as not beingsuitable for super-resolution imaging.
 26. The method of claim 25,wherein cross-correlating the second artifact to the known artifactsthat have been assessed as not being suitable for super-resolutionimaging comprises: measuring a similarity of the second artifact to eachknown artifact as a function of a displacement of the second artifactrelative to each known artifact.
 27. The method of claim 21, furthercomprising: determining, by the computing system, a total number ofartifacts on the specimen based on the single coherent image.
 28. Anon-transitory computer readable medium comprising one or more sequencesof instructions, which, when executed by one or more processors, causesa computing system to perform operations comprising: obtaining, by thecomputing system, a low resolution image of a specimen using a lowresolution objective of a microscopy inspection system; detecting, bythe computing system, a plurality of artifacts in the low resolutionimage; determining, by the computing system, that a first artifact ofthe plurality of artifacts is suitable for super-resolution imaging;determining, by the computing system, that a second artifact of theplurality of artifacts is not suitable for super-resolution imaging; andbased on the determining, generating, by the computing system, asuper-resolution image of a first portion of the specimen that includesthe first artifact, generating, by the computing system, a highresolution image of a second portion of the specimen that includes thesecond artifact, and generating, by the computing system, a singlecoherent image of the specimen by combining the super-resolution imageand the high resolution image.
 29. The non-transitory computer readablemedium of claim 28, wherein determining, by the computing system, thatthe first artifact of the plurality of artifacts is suitable forsuper-resolution imaging comprises: cross-correlating the first artifactto known artifacts that have been assessed as suitable forsuper-resolution imaging.
 30. The non-transitory computer readablemedium of claim 29, wherein cross-correlating the first artifact to theknown artifacts that have been assessed as suitable for super-resolutionimaging comprises: measuring a similarity of the first artifact to eachknown artifact as a function of a displacement of the first artifactrelative to each known artifact.
 31. The non-transitory computerreadable medium of claim 29, wherein the known artifacts that have beenassessed as suitable for super-resolution imaging comprises artifactswhere super-resolution images were generated and determined to behigh-confidence super-resolution images.
 32. The non-transitory computerreadable medium of claim 28, wherein determining, by the computingsystem, that the second artifact of the plurality of artifacts is notsuitable for super-resolution imaging comprises: cross-correlating thesecond artifact to known artifacts that have been assessed as not beingsuitable for super-resolution imaging.
 33. The non-transitory computerreadable medium of claim 32, wherein cross-correlating the secondartifact to the known artifacts that have been assessed as not beingsuitable for super-resolution imaging comprises: measuring a similarityof the second artifact to each known artifact as a function of adisplacement of the second artifact relative to each known artifact. 34.The non-transitory computer readable medium of claim 28, furthercomprising: determining, by the computing system, a total number ofartifacts on the specimen based on the single coherent image.
 35. Asystem comprising: a processor; and a memory having programminginstructions stored thereon, which when executed by the processor,causes the system to perform operations comprising: obtaining a lowresolution image of a specimen using a low resolution objective of amicroscopy inspection system; detecting a plurality of artifacts in thelow resolution image; determining that a first artifact of the pluralityof artifacts is suitable for super-resolution imaging; determining thata second artifact of the plurality of artifacts is not suitable forsuper-resolution imaging; and based on the determining, generating asuper-resolution image of a first portion of the specimen that includesthe first artifact, generating a high resolution image of a secondportion of the specimen that includes the second artifact, andgenerating a single coherent image of the specimen by combining thesuper-resolution image and the high resolution image.
 36. The system ofclaim 35, wherein determining that the first artifact of the pluralityof artifacts is suitable for super-resolution imaging comprises:cross-correlating the first artifact to known artifacts that have beenassessed as suitable for super-resolution imaging.
 37. The system ofclaim 36, wherein cross-correlating the first artifact to the knownartifacts that have been assessed as suitable for super-resolutionimaging comprises: measuring a similarity of the first artifact to eachknown artifact as a function of a displacement of the first artifactrelative to each known artifact.
 38. The system of claim 35, whereindetermining that the second artifact of the plurality of artifacts isnot suitable for super-resolution imaging comprises: cross-correlatingthe second artifact to known artifacts that have been assessed as notbeing suitable for super-resolution imaging.
 39. The system of claim 38,wherein cross-correlating the second artifact to the known artifactsthat have been assessed as not being suitable for super-resolutionimaging comprises: measuring a similarity of the second artifact to eachknown artifact as a function of a displacement of the second artifactrelative to each known artifact.
 40. The system of claim 35, wherein theoperations further comprise: determining a total number of artifacts onthe specimen based on the single coherent image.